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The National Household Education Survey (NHES) is a data 
collection system of the National Center for Education Statistics (NCES) 
designed to address a wide range of education-related issues. It is a 
telephone survey of the noninsti tutionalized civilian population. In 1995, 
the NHES included a survey of adult education (AE) . This report deals with 
response bias in the AE component of the NHES. To estimate response bias for 
the AE, an intensive reinterview was conducted with a sample of respondents 
and the responses to this reinterview were compared with the original values 
to estimate response bias. Interviewers were not chosen from the regular pool 
of NHES interviewers, but were people with previous experience in less 
structured interviews . An interview protocol was developed for the 
reinterview. Of 230 cases sampled for the bias study, 206 respondents 
completed intensive reinterviews. The estimated bias in the overall 
participation rate of adults was 24%, and the bias-corrected estimate is that 
64% of adults participated in AE in 1995, a figure considerably larger than 
the 40% reported in the NHES: 95. Under-reporting may have resulted from a 
lack of understanding related to the less formal forms of adult education. 

The under-reporting of participation for work-related and personal 
development courses was substantial. Consistency problems were identified for 
items dealing with barriers to participation in adult education, but the 
reinterview program did not lead to new ideas to cope with inconsistent 
reporting (the switching problem) in the NHES: 95. The intensive reinterview 
methodology appeared to have good potential for detecting biases, and it 
appears that response bias can be estimated using this methodology. (Contains 
9 tables and 14 references.) (SLD) 
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Background 



The National Household Education Survey (NHES) is a data collection system of the 
National Center for Education Statistics (NCES) designed to address a wide range of education-related 
issues. NCES has as its legislative mission the collection and publication of data on the condition of 
education in the Nation. The NHES is specifically designed to support this mission by providing 
information on educational issues best addressed by contacting households rather than schools or other 
educational institutions. The NHES provides descriptive data on the educational activities of the U.S. 
population and offers policymakers, researchers, and educators a variety of statistics on the condition 
of education in the U.S. 

The NHES is a telephone survey of the noninstitutionalized civilian population of the 
U.S. Households are selected for the survey using random digit dialing (RDD) methods and data are 
collected using computer-assisted telephone interviewing (CATI) procedures. Approximately 60,000 
households are screened for each administration, and individuals within households who meet 
predetermined criteria are sampled for more detailed or extended interviews. The data are weighted to 
permit estimates of the entire population. The NHES survey for a given year typically consists of a 
screener, which collects household composition and demographic data, and extended interviews on two 
substantive components addressing education-related topics. In order to assess data item reliability and 
inform future NHES surveys, each administration also includes a subsample of respondents for a 
reinterview. 1 



The primary purpose of the NHES is to conduct repeated measurements of the same 
phenomena at different points in time. Throughout its history, the NHES has repeatedly collected data 
in ways that permit estimates to be tracked across time. This includes repeating topical components on 
a rotating basis in order to provide comparative data across survey years. In addition, each 
administration of the NHES has benefited from experiences with previous cycles, resulting in 
enhancements to the survey procedures and content. Thus, while the survey affords the opportunity 
for tracking phenomena across time, it is also dynamic in addressing new issues and including 
conceptual and methodological refinements. 



See 1995 Adult Education Data File User’s Manual for additional information. 



A new design feature of the NHES program implemented in the NHES:96 is the 
collection of demographic and educational information on members of all screened households, rather 
than just those households potentially eligible for a topical component. In addition, this expanded 
screening feature includes a brief set of questions on an issue of interest to education program 
administrators or policymakers. The total screener sample size is sufficient to produce state estimates 
of household characteristics for the NHES:96. 

Full-scale implementations of the NHES have been conducted in 1991, 1993, 1995, and 
1996. Topics addressed by the NHES:91 were early childhood education and adult education. The 
NHES: 93 collected information about school readiness and school safety and discipline. The 1991 
components were repeated for the NHES:95, addressing early childhood program participation and 
adult education. Both components underwent substantial redesign to incorporate new issues and 
develop new measurement approaches. In the NHES:96, the topical components were parent/family 
involvement in education and civic involvement. In addition, the NHES:96 expanded screening feature 
included a set of questions on public library use. 



In addition to its topical components, the NHES system has also included a number of 
methodological investigations. These have resulted in technical reports and working papers covering 
diverse topics such as telephone undercoverage bias, proxy reporting, and sampling methods. This 
series of technical reports and working papers provides valuable information on ways of improving the 
NHES. 

2. Purpose and Overview 

2.1 Response Errors 

Estimates from surveys are subject to both variable and systematic nonsampling errors 
(Groves 1989). Variable nonsampling errors are those that might vary across repeated surveys 
administered to the same sample, assuming that the conditions of the interview could be controlled so 
that the surveys were independent. For example, the same respondent might report annual income 
differently when asked in repetitions of the same survey because the method used by the respondent to 
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estimate income might vary (records might be used, recall might be used, or the value might be 
estimated using different schemes). These circumstances would lead to variable errors for estimates of 
income. This type of error is often called response variance. 

Systematic nonsampling errors, on the other hand, are those that have a particular 
direction. For example, if respondents tend to omit certain types of income, say interest income from 
savings, then the estimated income would be expected to be lower than the true income. In repetitions 
of the same survey, the estimated income would always be less than the true income. These types of 
systematic errors are often called response bias. Survey estimates can be subject to both response 
variance and response bias. 

Reinterviews have been used to estimate the response variance for many estimates in 
previous NHES surveys (Brick and West 1992; Brick, Rizzo, and Wernimont 1996)! A reinterview 
was also conducted for the Adult Education (AE) component of the NHES: 95 for this purpose and the 
results of the reinterview are contained in a separate working paper (Brick, Wernimont, and Montes 
1996). The 1995 AE reinterview is referred to often in this report as the Reinterview Study. 

Methods used in the reinterviews, such as selecting interviewers from the original 
interviewer pool, asking the questions in much the same way as asked in the original interview, not 
informing the interviewer or the respondent of the answers from the original interview, and waiting at 
least 14 days between interviews so that the respondent will not remember the details of the original 
interview are all designed to support the measurement of response variance. Forsman and Schreiner 
(1991) discuss how these types of procedures can be used to make the reinterview circumstances as 
similar as possible to those of the original interview. 

Measuring response bias is more difficult. One way of measuring response bias is to 
compare the results of the survey against answers from a more definitive source, such as an 
administrative record file. This record check or validation approach has serious limitations of its own 
(Groves 1989). For example, record checks can only be used if records exist on the survey topic and 
those records can be accessed. Another key assumption is that the records are complete and accurate 
and can be matched to the survey respondents without error. Brick et al. (1994) found that these 
assumptions did not hold well even for the well-defined topic of teacher certification. 



Another way of measuring response bias is through the use of reinterviews. As noted 
above, reinterviews are ordinarily undertaken to measure response variance rather than response bias. 
However, sometimes a process called reconciliation is used in reinterviewing to measure bias. If the 
original and reinterview responses are different, then the respondent is asked to reconcile the 
differences and the resulting response is called the reconciled response. The reconciliation is often 
conducted by a supervisor rather than a regular interviewer to make the reconciled response less 
subject to error. The assumption that a supervisor is more likely to obtain and record the correct 
response is itself tenuous. Under this assumption, researchers have assumed the reconciled response to 
be more accurate, and then the difference between the original and reconciled response can be used to 
estimate response bias (Forsman and Schreiner 1991). 

Reconciliation has been used in previous NHES studies to estimate response bias (Brick 
and West 1992; Brick, Rizzo, and Wernimont 1996), but there is little evidence that it actually 
measured bias. The interviewers used were regular telephone interviewers and the estimates of 
response bias based on the reconciled responses were not very different from the responses based on 
the unreconciled reinterview data. Regardless of how these differences between the reinterview and 
reconciled reinterview responses are interpreted, the reconciliation produced little additional 
information. As a result of this, responses for only a few items were reconciled in the Reinterview 
Study and the goals of the reinterview did not include estimating response bias. 

An alternative method was chosen to estimate response bias for the AE component of the 
NHES:95 and is the basis of this research. An intensive reinterview was conducted with a sample of 
respondents and the responses to this reinterview are compared to the original values to estimate 
response bias. Throughout this report, this study is called either the Bias Study or the intensive 
reinterview. 



The intensive reinterview for the NHES:95 differed from the Reinterview Study in a 
number of ways. First, the interviewers were not selected from the pool of regular NHES 
interviewers, but were persons with previous experience in interviewing using less structured methods. 
The interviewers were trained to use a protocol and to conduct the reinterviews in a conversational 
mode, using probes and other devices to trigger recall and comprehension. The reinterview was 



focused on a few topics and ample time was allowed for discussing these few points. The respondents 
were encouraged to voice their opinions and understanding of the topics. Furthermore, attempts were 
made to engage the respondents in the interview by explicitly asking for their advice on ways to 
improve the interview. The hope was that these methods would lead to more complete and accurate 
reporting in the intensive reinterview. 

The intensive reinterview method was pioneered by Belson (1986) who focused on 
difficult or sensitive topics primarily in opinion and marketing research. He proposed and 
implemented many techniques in his research that are today classified as cognitive methods. In the 
Teacher Follow-up Survey (TFS) in the Schools and Staffing Survey, an extensive reconciliation was 
conducted using methods that had some of the same attributes of intensive reinterviews (Jenkins and 
Wetzel 1994). Their approach involved an extensive, structured reconciliation using supervisors as 
interviewers to try to identify problem questions and to offer suggestions for improving the 
questionnaires. 

The methods used in the NHES:95 Bias Study are most closely related to those proposed 
by Belson (1986). Four specific types of issues were identified prior to the study and methods were 
developed to address these topics, excluding other parts of the AE interview from this study. The 
methods used in the Bias Study, including training the interviewers, sampling respondents, and 
conducting the intensive reinterviews, are described in some detail in the following sections after the 
goals of the intensive reinterview are presented. 



2,2 Goals 

Four major research objectives were established for the Bias Study. They were: 

• To examine the bias in the overall participation rate in adult education activities; 

• To examine bias in the estimates of participation in work-related and personal 
development courses; 



To assess the validity of the responses to the barriers items; and 



• To explore reasons for switching participation status between the NHES:95 
screener and the AE extended interview. 

Each of these objectives is discussed below. 

Participation rates. The first goal of the Bias Study was to examine the potential bias 
due to either underreporting participation or overreporting activities that took place outside of the time 
frame of the survey (i.e., the past 12 months prior to the original NHES interview). Respondents 
might underreport participation either because they might not recall a qualifying activity during the 12 
months before they were interviewed or because they might not comprehend the range of activities that 
were included as adult education. These types of underreporting would lead to downward bias in 
participation rates. However, an upward bias could occur if respondents "telescoped" some activities. 
Telescoping is reporting activities that took place outside of the time frame of the survey as having 
taken place within that time frame (Neter and Waksberg 1964). If their only activities were outside of 
the time frame of the NHES and they were reported as having taken place in the past 12 months prior 
to the original interview, participation rates would be biased upward. Underreporting was expected to 
be minimal in all types of adult education (the six types of activities were: English as a Second 

Language (ESL), adult basic education/GED preparation classes (ABE/GED), credential programs, 
apprenticeships, work-related courses, and personal development courses), except work-related and 
personal development courses. As a result, the Bias Study focused solely on these two types of 
participation in order to assess the bias in the overall participation rate. 

Participation rates in work-related and personal development courses. The second 
goal was to obtain more accurate estimates of participation in work-related and personal development 
courses, separately. As a result of the differences between the estimates of participation from the 
NHES:91 and the previous CPS data collections (i.e.. Adult Education supplements, 1969-1984), it 
was suspected that work-related courses and personal development courses were susceptible to 
underreporting. The cognitive laboratory work on the NHES:95 questionnaire reinforced this 
impression. One of the major concerns for reporting these types of courses is that respondents might 
not comprehend the full range of activities that are included as work-related and personal development 
courses. Some respondents might have their own definitions of adult education that are not consistent 
with those of the survey. These types of comprehension problems could combine with recall problems 
and result in underreporting of work-related and personal development courses . 



Barriers to participation. The responses to the barriers questions, in particular those 
asked of low-education (i.e., less than a high school diploma or GED) nonparticipants, are important 
from a policy perspective, but concerns about the validity of these items were raised during the design 
phase of the survey. Nonparticipants were asked several questions related to their not having 
participated in ESL, ABE/GED, or work-related courses. They were asked whether they were 
interested in participating in adult education activities, and, if they were, whether they knew about 
activities in which they could participate. Those who were interested and knew about activities were 
asked whether time, money or cost, child care, transportation, or any other barrier had prevented them 
from participating. In order to assess the efficacy of the barriers items in the NHES:95 interview, the 
Bias Study further investigated the barriers to participation in ESL, ABE/GED, or work-related 
courses that low-education nonparticipants perceived. 

Switching participation status. When the household members were enumerated during 
the screening interview in the NHES, questions were asked to determine eligibility for sampling 
persons for more detailed or extended interviews. In the NHES:91 and the NHES. 95, the household 
respondent answering the screening items was asked whether or not the adults in the household had 
participated in AE activities in the last 12 months and their responses were used to assign sampling 
rates to the adults. In both surveys, a considerable number of adults who were sampled and completed 
the extended interview reported their participation status differently than was reported in the screening 
interview. 



One possible reason for switching is that the household respondent answering the 
screening questions might not have the knowledge necessary to accurately report the participation 
status of the adult. This reason would only affect those adults who did not respond to both the 
screening and extended interviews themselves. Another possible reason for switching was that the 
screening question was very simple and respondents may not have fully understood what was meant by 
participation in adult education. The extended interview had a series of more detailed questions that 
asked about specific types of activities. Both of these reasons for switching were investigated. 

While this switching does not result in bias in the estimates, it does reduce the efficiency 
of the sampling and increase the sampling errors of the estimates. Therefore, one of the goals of the 



Bias Study was to better understand the switching phenomenon and obtain qualitative information on 
the causes of it. 



3. Intensive Reinterview Method 

The method used in the Bias Study to obtain a more accurate value for selected 
characteristics in the NHES:95 AE was an intensive reinterview. In an attempt to more closely 
determine the respondent's actual status or opinions, the intensive interview was more of a directed 
conversation between the respondent and the interviewer rather than a formally scripted interview. 
Respondents were reminded of their answers in the original survey and asked if the answers were still 
true for them. They were asked to recall other details related to their responses. Interviewers were 
fully knowledgeable about the original answers given by the respondent. Tactics similar to those used 
in cognitive laboratory work, such as asking open-ended questions and using probes to encourage the 
respondent to elaborate on his or her answer, were used. The goal was to obtain more detailed and 
accurate information by understanding the respondent's perspective and the reasons for his or her 
answers. 



Although cognitive laboratory research has been a method used in the design phase of 
each NHES administration (Nolin 1996), the intensive reinterview was a new undertaking and 
presented several challenges that could have affected the responses. For example, the interviewers 
who conducted the NHES interviews have been thoroughly trained to read the questions verbatim and 
to avoid affective behavior that might influence the respondent. Adopting the conversational and 
unstructured interviewing method called for in the intensive reinterview requires major changes in their 
behavior. The interviewers were also called upon to implement some methods used in cognitive 
research, but they were not previously trained in these methods. The respondents also faced a 
challenge because the intensive reinterview differed significantly from the type of interview they had 
already done. They were called upon to give reasons for their responses and provide details rather 
than choose among response alternatives. To address these challenges, a protocol and data collection 
methods were developed especially for this study. 



3.1 



Protocol Development 



The protocol developed for the Bias Study contained information on the respondent's 
participation status, education level, and responses in the original survey, along with a few open ended 
questions and suggested techniques for eliciting detailed answers. One interviewer was trained and 
then conducted several interviews that were monitored by Westat and NCES staff. These initial 
interviews served as a pilot test. Minor revisions of the protocol were made based on this experience 
before the majority of intensive reinterviews were conducted. The Adult Education Bias Study 
protocol is found in Appendix A. 

Two difficulties encountered during the pilot training and interviewing were remedied in 
the revised protocol. First, the interviewer relied too heavily on the comments and probes suggested 
in the protocol. They were intended to be a point of departure for the training discussions and a 
suggestion for the interviewers to obtain the in-depth responses that were sought. However, because in 
most studies interviewers are required to read scripts verbatim, it took some encouragement and 
several practice interviews before the conversational approach was employed adequately. 

In the revision of the protocol, the initial questions and suggested probes were reduced in 
number and reworded into a more conversational tone. Interviewers, thus, had to rely more on their 
own words, or, if they did read verbatim, at least the tone was more conversational. To prompt recall 
of courses, the protocol also contained examples of educational activities. These were simplified and 
reorganized by type (hobby/special interest, personal development, work-related) so that interviewers 
could mention some of each type to nonparticipants and some of the type reported previously by 
participants. This was an effort to promote recall. During the second phase of training, interviewers 
were asked to think of additional probes for each topic in the protocol and make note of ones they 
would be most comfortable using. They received feedback during the practice interviews according to 
their skill in adopting a conversational tone and using probes that arose from interaction with the 
particular respondent. 

The other difficulty with the original protocol involved the use of a timeline that had been 
designed to help respondents remember key events and properly date them. Creating and using the 
timeline was intended to aid respondent recall about participation in adult education activities and to 
test for telescoping. It was suggested that meaningful dates, e.g., birthdays, holidays, or seasons be 



elicited from the respondent and noted on the timeline for use in bounding discrete time intervals and 
asking respondents about each short interval. The first interviewer found this procedure awkward. 
The respondents did also because the task was out of context for them. 

The protocol was revised to include a timeline with major dates that the interviewer could 
use for reference, if he or she wished. No overt explanation was made of the concept of the timeline 
to the respondent, who was simply urged to think back over the time prior to the date of the original 
interview. Interviewers were told to use the respondent's own time context, if possible, to aid recall. 
For example, if a respondent had reported that he or she had taken a course in the spring, the 
interviewer probed for activities during the summer or fall. Or, if the respondent had taken a course 
right after the New Year, he or she was asked about participating in activities between Thanksgiving 
and the Christmas holidays. 



3.2 Interviewer Selection and Training 

The intensive reinterviews were conducted by five interviewers who were carefully 
selected for the study. The interviewers were not members of the regular pool of NHES interviewers. 
The interviewers for the Bias Study were selected from a specially trained pool of interviewers who 
had experience in a wide variety of educational studies and previous experience in conducting semi- 
structured interviews. These interviewers were able to frame their next question to match the 
respondent’s previous answers. 

Interviewer training was conducted with two primary goals. The first goal was to 
thoroughly acquaint the interviewers with the purpose of the Bias Study and the methodology that 
would be used. The second was to model the behavior that the interviewers were to adopt in order to 
conduct the intensive reinterviews. So, although a training guide was developed (Appendix B), 
detailed written training materials were not distributed to the interviewers. Each interviewer received 
a handout (Appendix C) with the goals of the study and key points about the methodology. The 
training was conducted as a dialogue between the trainer and the interviewers. The interviewers were 
encouraged to ask questions, verify their understanding of the points made, and offer whatever 
suggestions or critiques they thought appropriate. 



A three hour training session, which was attended by Westat and NCES staff, began with 
an overview of the purpose of the Bias Study, a brief explanation of the methodology, and a 
presentation on the background of the AE component. Each of the four goals of the study was 
explained and the characteristics of the population that would be responding to each section of the 
questionnaire were discussed. The purpose of the intensive reinterview and the techniques designed to 
elicit the information were thoroughly reviewed. Interviewers were instructed in active listening, 
giving neutral feedback to encourage in-depth responses, probing for details, and the creative use of 
silences. They were encouraged to use probes that reflected the meanings and the context that were 
expressed by the respondent. As the interviewers reviewed each section of the protocol with the 
trainer, they were reminded to engage in conversation with the respondent, paying close attention to 
what he or she was saying, and recording only key words while on the telephone; other comments 
were to be recorded immediately after ending the call. Six role plays were also included in the 
training, with a lengthy debriefing after each. 

The first interviews were monitored, and the protocol revised, as explained above. The 
next set of interviews was conducted by the first interviewer after training on the revised protocol. 
These interviews were monitored by the trainer and the other interviewers. The entire group then 
discussed the techniques they observed and suggested other ways of obtaining information from the 
respondents. This served as additional training for the interviewers. Monitoring of interviews 
continued as an additional check on the efficacy of the protocol and the skill of the interviewers. 
Interviewers were provided with feedback from the trainer on engaging in purposeful conversation with 
the respondent and using appropriate probes. 



3.3 Sampling 

The main goal of the Bias Study was to develop estimates of bias, however, the budget 
provided for a very limited sample size. Because of the small sample sizes, it was decided that the 
typical design-based estimates gathered from the original interview would be subject to very large 
sampling errors and relationships would be obscured by these sampling errors. Thus, the results from 
this relatively small sample were analyzed assuming the observations were from independent, 
identically distributed random variables and sampling weights were not used. This assumption 



naturally leads to relatively small sampling errors for the estimates of bias. These estimates are only 
used to explore the potential bias in the participation rates and cannot be extrapolated to the universe of 
all adults. This point is emphasized in the final section of the report. 

The sample was randomly selected from both participants and nonparticipants who 
completed the AE extended interview and were not included in any other special studies (i.e., 
NHES:95 Splice Sample Interview, NHES:95 Expanded Screener Interview, and Reinterview Study). 
In order for a case to be eligible for the study, the following conditions had to be met: 

• The original interview was never a refusal or a language problem; 

• The original interview was conducted in English (as opposed to Spanish); 

• Only one interview in a household was eligible; and 

• All other extended interviews in the household had been completed. 

The eligible cases were first classified into the following non-exclusive, six groups for 

sampling: 

1 . Adults who completed the original interviews as participants in adult education; 

2. Adults who were sampled as nonparticipants but completed the interviews as 
participants in adult education; 

3. Adults who were sampled as participants but completed the interviews as 
nonparticipants in adult education; 

4. Adults who were sampled as low-education nonparticipants and answered the 
questions about obstacles to taking courses; 

5. Adults who were sampled as low-education nonparticipants and did not answer the 
questions about obstacles to taking courses; and 

6. Adults who were sampled as high-education nonparticipants or participants. 

Since adults could belong to more than one group (e.g., they could have switched from 
participation to nonparticipation status, group 3, and been sampled as high education participants, 
group 6), some eligible adults had more than one chance of being sampled for the study. The sample 
was drawn by randomly selecting adults from group 1, then sampling all eligible adults in group 2, 



provided they were not already sampled for the study from group 1 . The sampling proceeded in this 
way until a sample of 230 adults was selected from all 6 groups, with 48, 27, 25, 34, 42, and 54 
sampled from the groups, respectively. The overall sample size was derived based on the budget 
available and on discussions with NCES with respect to the numbers in each group. 

Of the 230 cases sampled for the Bias Study, 206 respondents (90 percent) completed 
intensive reinterviews. Table 1 shows the characteristics of the 206 respondents by sampling group, 
participation status, and educational level. The respondents to the Bias Study had characteristics 
consistent with the targets developed for sampling. 

Of the 24 cases not completed, there were 5 "refusal" cases (i.e., respondents refused to 
participate in the Bias Study interview), 4 cases in which telephone numbers were either disconnected, 
non-working, or changed to an unpublished number, and 14 cases in which the telephone was never 
answered (including 5 that were answered only by answering machines). 



Table 1.-- Characteristics of respondents of the intensive reinterview, by sampling group, participation 
status, and educational level 



Group 


Total 


Nonparticipant 


Participant 


Became nonparticipant 


Became participant 


Low 

education 


High 

education 


Low 

education 


High 

education 


Low 

education 


High 

education 


Low 

education 


High 

education 


Total 


206 


38 


34 


23 


38 


14 


29 


5 


25 


1 


41 






4 


37 










2 


26 














2 


24 


3 


23 










8 


15 






4 


27 


9 




13 




3 




2 




5 


41 


29 


1* 


6 


1* 


3 




1 




6 


48 




33 








14 




1 



Note: Low education = less than a high school diploma or GED; high education = high school diploma or higher. 

♦Sampled as low-educated adults (i.e., adults without a high school diploma) in the screener interview, but reported that they 
had a high school diploma or higher education in the extended interview. 

Source: U.S. Department of Education, National Center for Education Statistics, Bias Study of the National Household 
Education Surv ey , 1 995 . 



3.4 



Data Collection 



Data collection for the Bias Study began 10 weeks after the beginning of data collection 
for the full survey and extended over 6 weeks. Because all cases that met the four criteria for 
eligibility were included in the pool for sampling, some respondents were recontacted shortly after the 
original interview and some after a longer time. The time interval between the original interview and 
the intensive reinterview was between 6 and 18 weeks. As Forsman and Schreiner (1991) discuss, the 
time between interviews could have some effect on the estimates, but not much research has been done 
on this topic and it was not considered in this study. 

Interview protocols were prepared by project staff and reviewed by interviewing staff 
prior to placing telephone calls. Each protocol was customized for the particular respondent and 
contained his or her name and telephone number, level of education, participation status, whether 
interest or barriers questions had been answered in the original survey, and whether participation status 
had switched during the interview. Thus, before talking with the respondent, the interviewer was 
thoroughly acquainted with his or her original responses to the items included in the intensive 
reinterview. Throughout data collection, the completed forms were monitored by project staff to 
ensure that all information was recorded appropriately. 

During the data preparation, the information on the courses reported by respondents 
during the intensive reinterview were examined thoroughly. For example, project staff reviewed the 
time period of all courses reported in the Bias Study to make sure that they were taken during the 12- 
month period prior to the original interview. Project staff first verified that each course reported in the 
intensive interview was not reported in the original interview. Based on course names, each newly 
reported course was classified into one of three categories: work-related, personal development, and 
indeterminable. 

In the Bias Study, there were five cases in which some or all of the new course names 
reported were the same (or very probably the same) as activities reported in other sections of the 
original interview, i.e., credential courses, ESL courses, or GED courses. Those courses were deleted 
from the analysis. 



4. 



Findings 



The data collected in the Bias Study were analyzed to address each of the four major 
objectives of the study. First, the potential biases in the estimates of the overall AE participation rate 
(based on the questions about participation in either work-related or personal development courses) and 
the separate participation rates for work-related and personal development courses are described along 
with interpretations of the likely reasons for the bias and suggestions on ways to reduce the bias. The 
issue of the validity of estimates based on the barriers items is addressed next. Finally, a qualitative 
analysis of the reasons for adults changing participation status is presented. 



4.1 Bias in Participation Rate Estimates 

The first and arguably the most important goal of the Bias Study was to estimate the 
response bias associated with estimates of the rate of participation in AE from the NHES.95. Adults 
were classified as participants in AE if they had participated in one or more of six different types of 
adult education activities during the past 12 months. The six types of activities were: ESL, 
ABE/GED, credential programs, apprenticeships, work-related courses, and personal development 
courses. Based on the responses in the NHES:95, 40 percent of all adults had participated in one or 
more of these activities in the last 12 months (Kim, Collins, Stowe, and Chandler 1995). 

As noted earlier, responding to the items about participation in work-related and personal 
development courses was identified as potentially being problematic. After phase 1 of the field test, 
interviewers reported that the participation questions in the work-related and other structured sections 
were too long, and some respondents expressed impatience when listening to the introductions. The 
length of these items resulted from the desire to provide a sufficient number of examples to prompt the 
respondent’s memory. As a result of this reaction and monitoring the interviews, shorter questions for 
both sections were developed. Phases 2 and 3 demonstrated that the new introductions to these 
sections were significantly improved. 



The participation rate of 40 percent excludes adults who participated only in full-time credential programs. 



After the survey was completed, the results from the Reinterview Study confirmed that 
these two types of participation were much more likely to be reported inconsistently than any of the 
other types (Brick, Wernimont, and Montes 1996) even though the results were not statistically 
significant. These findings support the decision to restrict the Bias Study to an in-depth examination of 
reporting work-related and personal development courses. 

Adults were classified as nonparticipants in the original interview if they said they had 
not taken any courses in the last 12 months. Of the 115 nonparticipants who responded to the 
intensive reinterview, 41 percent (47) indicated that they had taken one or more work-related or 
personal development courses (in the intensive reinterview, respondents were not asked about other 
types of courses). Since none of the participants sampled for the intensive reinterview denied having 
taken courses or indicated they were not taken in the appropriate time period, the response bias in the 
overall participation rate is one-directional and substantial. 

Assuming the responding nonparticipants in the Bias Study are a simple random sample of 
all adults classified as nonparticipants in the NHES:95 (the analysis is thus unweighted), the bias in the 
NHES:95 estimate is 24 percent. 3 The bias is estimated by multiplying the percent of all adults who 
were nonparticipants as reported in the NHES:95 (60 percent) by the percent of the nonparticipants 
who reported participating in the intensive reinterview (41 percent). In general, the estimated bias is 

HPo ) = {Po~ I00)y np + Po9 p (4. 1) 

where p 0 is the estimate of the percentage of adults classified as participants in the initial interview, y p 
is the estimate of the proportion of participants in the initial interview who reported not participating in 
the intensive reinterview, and y np is the estimate of the proportion of nonparticipants in the initial 
interview who reported participating in the intensive reinterview. In this case, the last term of the 



The bias study addressed work-related and personal development courses. As a result, the adjusted rate does not reflect any changes in 
basic skills, ESL, credential, or apprenticeship programs. The reinterview found very low rates of gross and net differences for these types 
of AE, in fact, no differences for ESL. Their incidence is also low compared to work-related or personal development. As a result, these 
types of AE would not have a significant impact on bias adjusted rates. 



estimated bias is zero (y =0), because no initial participants said they had not taken any courses 
during the intensive reinterview. 

If the bias in the estimated percentage of adults who participated in AE is 24 percent, 
then the bias-corrected estimate is that 64 percent of adults participated in AE in 1995. This is 
substantially larger than the 40 percent reported in the NHES:95. Both the bias and the percentage 
participating from the NHES:95 are subject to sampling error and because of the small sample size the 
sampling error of the bias is very large relative to that for the estimate from the NHES:95. Taking 
advantage of the fact that y P =0, the estimated bias can be written as 

b(Po) = {Po ~ I00)y np 

Thus, the estimated bias is a product of random variables and the approximate variance for a product 
of independent random variables is (Hansen, Hurwitz, and Madow 1953) 

Var(b) = (y np ^ 2 Var(p 0 ) + (p Q - 100) 2 Var(y np) j (4.2) 

The approximate variance of the estimated bias is computed by substituting for the 
quantities in (4.2). The value of p 0 is 40 percent with an estimated variance of 0.25 percent (Kim, 
Collins, Stowe, and Chandler 1995) and the value of y np from the intensive reinterview is 0.41 (jjj) 

with an estimated variance of 0.002 (fij {ni(l “ Hi)}) • Substituting these values into (4.2) and 

taking the square root, the standard error of the estimated bias of 24 percent is 2.7 percent. Thus, a 
95 percent confidence interval for the estimated bias is from 19 percent to 29 percent, and for the 
percent of adults participating, the confidence interval is from 59 to 69 percent. The entire interval is 
considerably larger than the estimated 40 percent from the NHES:95. 

Considering the nonparticipating respondents to the intensive reinterview a simple random 
sample of all nonparticipants in the original survey is a key assumption in estimating the response bias. 
Usually, the sampling procedures would ensure that this assumption holds, but the sampling methods 
described earlier were primarily concerned with making sure the sample sizes for specific groups were 



large enough to provide enough adults with various characteristics. In addition, the overall small 
sample size in the Bias Study, and for the nonparticipant sample in particular, does not allow for broad 
generalizations. These estimates should be understood to be exploratory and do not permit making 
bias corrections to the full sample estimates. 

One way to evaluate this assumption is to compare characteristics of the responding 
nonparticipants from the Bias Study to the characteristics of all nonparticipants from the NHES:95. 
Table 2 shows the age, sex, educational attainment, labor force status, and race/ethnicity percentage 
distributions for the NHES:95 and Bias Study nonparticipants. The estimates for the NHES:95 use the 
appropriate weights, while the Bias Study are unweighted percentages. 



Table 2.— Characteristics of nonparticipants from the NHES:95 and the intensive reinterview 





NHES:95 




Total (in 1 ,000’s) 


Percent 


Bias Study 
percent 


Number of nonparticipants 


105,552 


100.0% 


100.0% 


Age 








Less than 30 


18,022 


17.1 


20.0 


31-50 


39,604 


37.5 


40.0 


51 or older 


47,896 


45.4 


40.0 


Sex 








Male 


51,957 


49.2 


54.8 


Female 


53,564 


50.8 


45.2 


Educational attainment 








Less than high school 


24,946 


23.6 


39.1 


High school diploma or GED 


59,076 


56.0 


40.0 


Associate's degree 


3,865 


3.7 


5.2 


Bachelor's degree or higher 


17,636 


16.7 


15.7 


Labor force status 








Employed 


53,652 


50.8 


50.4 


Unemployed 


4,305 


4.1 


7.0 


Not in labor force 


47,565 


45.1 


42.6 


Race/ethnicity 








White 


79,002 


74.9 


70.4 


Black 


12,105 


11.5 


16.5 


Hispanic 


9,830 


9.3 


7.8 


Other races 


4,585 


4.3 


5.2 



The number of nonpartici pants in the Bias Study was 115. 



Source: U.S. Department of Education, National Center for Education Statistics, Bias Study of the National Household Education Survey, 
1995 . 



The distributions of the nonrespondents for the NHES:95 and Bias Study are similar for 
most characteristics and the simple random sampling assumption does not seem to be a problem. The 
educational attainment distributions, on the other hand, are different, with a much larger percentage of 
the Bias Study nonparticipants having less than a high school education. This is not surprising, since 
low education adults were sampled at different rates for the Bias Study. Nevertheless, this does point 
out the fact that the Bias Study sample did have different sampling rates and these are not reflected in 
the unweighted distributions. It also highlights the fact that other characteristics correlated with 
educational attainment are not likely to be distributed in the same way in the Bias Study as in the 
NHES:95. The result is that it is not prudent to place too much reliance on the point estimates or 
confidence intervals from the Bias Study. They are subject to a specification error that cannot be 
measured. 



Even if the size of the estimated bias is not measured very precisely, the findings clearly 
show that a relatively large fraction of the adults classified as nonparticipants in the original survey 
take part in AE activities as reported in the intensive reinterview. Experience in other reinterviews 
(Brick and West 1992; Jenkins and Wetzel 1994) indicates that respondents are not very precise when 
asked for specific reasons for differences in their responses from one interview to the next. Because of 
this, respondents were not asked to report specific reasons for differences in their responses. 
However, it is likely that the survey methods are at least partially responsible for respondents reporting 
greater levels of participation in the intensive reinterview. In particular, providing more examples of 
courses, prompting the respondents to take time to think about their activities in the last year before 
responding, and specifically instructing the respondents that they should not limit themselves to formal 
courses in the intensive reinterview are potential reasons for the increased reporting. These potential 
reasons are examined in more detail later. 

In addition to nonparticipants, participants in the original survey who were sampled for 
the Bias Study were asked if there were any courses they had not reported in the initial interview by 
using the same types of probes described above. The participants were also asked to specify when they 
took these courses so it could be verified that the courses were taken in the 12 months prior to the 
original interview. The respondents were encouraged to verbalize how they knew when they took the 



course as a quality check and potential probe for additional courses. As a result of these questions, 
some participants identified additional courses that were not originally reported. All of the work- 
related and personal development courses participants reported in the NHES:95 were verified as being 
within the eligible 12 month time period. 

Table 3 shows the percentage of participants by the number of additional courses they 
reported in the intensive reinterview. The number of courses nonparticipants in the initial survey 
reported in the intensive reinterview are also shown in Table 3. In the original interview, about one- 
third of those who reported participating in adult education courses reported additional courses during 
the intensive reinterview. The average number of additional courses reported by participants and 
nonparticipants were approximately the same (1.6 courses for participants and 1.9 for nonparticipants. 



Table 3.— Number of adults reporting additional courses in the intensive reinterview, by participation 
status in the NHES:95 and number of additional courses 





Total 


Participant 


Nonparticipant 


Number 


Percent 


Number 


Percent 


Number 


Percent 


Total 


206 


100.0% 


91 


100.0% 


115 


100.0% 


Number of additional 














courses 














None 


130 


63.1 


62 


68.1 


68 


59.1 


One 


38 


18.4 


18 


20.0 


20 


17.4 


Two 


24 


11.7 


7 


7.7 


17 


14.8 


Three 


9 


4.4 


3 


3.3 


6 


5.2 


Four 


5 


2.4 


1 


1.1 


4 


3.5 



Note: Details may not add to 100 percent due to rounding. 

Source: U.S. Department of Education, National Center for Education Statistics, Bias Study of the National Household 
Education Survey, 1995. 

The reporting of additional work-related and personal development courses by adults 
classified as participants in the original survey is a further indication that the respondents may have 
had a more restrictive understanding of the scope of activities than was intended. Drawing on the 
work of Schwarz (1995), one interpretation of this finding is that respondents might have reacted to the 
context of the original interview in determining what was an eligible activity. The NHES:95 interview 



began by asking about more formal or traditional types of participation (ESL, ABE/GED preparation 
classes, credential programs, and apprenticeships) and some respondents may have created a response 
paradigm before the questions about the less formal activities (work-related and personal development 
courses) were asked. In the intensive reinterview, the context was different because the only types of 
courses discussed were work-related and personal development courses. If this hypothesis is correct, 
then modifying the interview at the juncture between formal and informal courses to make respondents 
more aware of the shift to less formal courses might improve reporting in future surveys. Other 
methods such as providing more examples and probes traditionally have been used for this purpose, 
but evidence described below suggests that more examples may not be very effective. 



4.2 Bias in Work-Related and Personal Development Participation Estimates 

Because such a large proportion of adults reported additional courses in the intensive 
reinterview, it is important to understand the types of courses they added and what this implies about 
the bias in the estimates of the participation rates for work-related and personal development courses. 
Below, the types of additional courses reported are examined first and then the bias in the participation 
rates are estimated. 

Table 4 shows the percentage of adults who reported additional work-related and personal 
development courses in the intensive reinterview. Because some adults reported both types of 
additional courses, the percentages across types sum to more than 100 percent. The bases for 
computing these percentages are given in the first row of the table, and include only adults who 
reported additional courses rather than all adults in the intensive reinterview. 



Table 4.-- Percentage of adults reporting additional courses in the intensive reinterview, by 
participation status and type of course added 





Total 


Participant 


Nonparticipant 


Number of adults adding courses 
Type of course 


76 


29 


47 


Work-related 


48.7% 


34.5% 


57.4% 


Personal development 


51.3 


65.5 


42.6 


Indeterminable 


7.9 


3.4 


10.6 



Note: Percentages add to more than 100 because adults may have reported additional work-related and personal development courses. 

These adults are included in both types of courses. 

Source: U.S. Department of Education, National Center for Education Statistics, Bias Study of the National Household Education Survey 

1995. 



Overall, about half the adults who named additional courses reported work-related 
courses and half-reported personal development courses. Participants were more likely to add personal 
development courses and the nonparticipants were more likely to add work-related courses, but these 
differences are not statistically significant. 

This phenomenon can also be examined by profiling the characteristics of adults who 
added courses. Table 5 gives the percentage distributions for adults who added courses by the type of 
courses added and the characteristics of the adults. The first column of the table gives the estimate 
from the NHES:95 for comparison purposes. In this table, persons who added both work-related and 
personal development courses are included in both of the last two columns of the table, while those 
who added only courses that could not be classified (indeterminable from Table 4) are excluded from 
the last two columns ot the table. As a result, the sum of the number of persons in the last two 
columns of the table adds to the total (76) merely by chance. The estimates give some indication of the 
types of response differences that may be present. For example, the results suggest that women were 
more likely to add personal development courses and men were more likely to add work-related 
courses. However, these differences are not very large or statistically significant. The last 
characteristic in the table is the overall participation status of the adult when sampled and when the 
interview with the adult was completed. For example, the ‘became nonparticipant’ row of the table 



refers to the adults who were sampled as participants (as designated by the person who answered the 
screening interview) and, when interviewed, the adult reported no adult education participation. The 
last two rows are based on very small sample sizes, but from the participant and nonparticipant rows it 
is clear that participants and nonparticipants were likely to add work-related and personal development 
courses at different rates. 



Table 5. --Percentage of adults reporting additional courses in the intensive reinterview, by type of 
course added and characteristic of the adult 



Characteristic 


NHES:95 


Bias Study 


Total 

(in 1 ,000’s) 


Percent 


Total 


Added 

work-related 


Added 

personal 

development 


Number of adults 


189,576 


100.0% 


76 


37 


39 


Age 












Less than 30 years 


46,159 


24.3 


22.4% 


21.6% 


23.1% 


31 to50 years 


80,091 


42.3 


53.9 


54.1 


53.8 


51 years or older 


63,327 


33.4 


23.7 


24.3 


23.1 


Sex 












Male 


90,275 


47.6 


50.0 


56.8 


43.6 


Female 


99,301 


52.4 


50.0 


43.2 


56.4 


Educational attainment 












Less than a high school diploma 


29,699 


15.7 


25.0 


29.7 


20.5 


High school diploma or GED 


103,367 


54.5 


38.2 


32.4 


43.6 


Associate's degree 


9,975 


5.3 


10.5 


8.1 


12.8 


Bachelor's degree or higher 


46,535 


24.5 


26.3 


29.7 


23.1 


Labor force status 












Employed 


117,833 


62.2 


71.6 


79.5 


68.3 


Unemployed 


8,167 


4.3 


10.8 


7.7 


12.2 


Not in labor force 


63,576 


33.5 


17.6 


12.8 


19.5 


Race/ethnicity 












White 


144,602 


76.3 


73.0 


64.1 


80.5 


Black 


20,808 


11.0 


12.2 


17.9 


7.3 


Hispanic 


15,705 


8.3 


8.1 


5.1 


9.8 


Other races 


8,461 


4.5 


6.8 


12.8 


2.4 


NHES:95 participation status 












Participant 


57,878 


30.5 


28.9 


18.9 


38.5 


Nonparticipant 


94,428 


49.8 


25.0 


27.0 


20.5 


Became participant 


26,176 


13.8 


11.8 


10.8 


12.8 


Became nonparticipant 


11,094 


5.9 


35.5 


43.2 


28.2 



Note: Percentages may add to greater than 100 because adults who added both work-related and personal development courses are included 
in both of the last two columns of the table. Persons who added only courses that could not be classified as either work-related or 
personal development are not included. 

Source: U.S. Department of Education, National Center for Education Statistics, Bias Study of the National Household Education Survey 
1995. 



Based on the NHES:95 responses, 21 percent of adults were estimated to have 
participated in work-related courses during the previous 12 months and 20 percent were estimated to 
have participated in personal development courses (Kim, Collins, Stowe, and Chandler 1995). The 



extent of the bias in these estimates can be estimated using equation (4.1). The bias for the work- 
related participation rate is 



kpo, W r) = (Po, W r-lW)y nP (4-3) 

where Pq wt * s the estimate of the percentage of adults classified as work-related participants in the 
initial interview, and y^ r is the estimate of the proportion of adults who did not report participating in 
work-related activities in the initial interview but reported participating in the intensive reinterview 
(since more of the participants denied having participated). 

The value of p 0 , wr is 21 percent from the NHES:95. Because we are now dealing with 
participation in a particular type of adult education, the value of y^r has two components: those 
classified as nonparticipants who reported taking work-related courses in the intensive reinterview, and 
participants in the initial survey who reported taking work-related courses for the first time in the 
intensive reinterview. In the Bias Study, 23 percent of the nonparticipants (27 of the 115) reported 
taking work-related courses and 8 percent (7 of 91) of the participants reported taking work-related 
courses for the first time. 



Substituting these into (4.3), the estimated bias for the percent of adults participating in 

/ ^ 27 2\ 7 \ 

work-related courses is 16 percent (21-100) J — : — h . The standard error can be 

v v.81 115 .81 9V 

computed using (4.2), where y^ r is treated as a sum of the two components described above and each 



of the components is itself a product of independent random variables. Using this approach, the 
standard error of the estimated bias is 3 percent and the 95 percent confidence interval for the 
estimated bias is from 10 to 22 percent. Translating to the estimated percentage of adults 
participating in work-related courses, the point estimate is 37 percent rather than the 21 percent 
estimated from the NHES:95 and the bias-corrected 95 percent confidence interval is from 31 percent 
to 43 percent. 



The same calculations can be performed for personal development courses to compute the 
estimated bias and its standard error. The quantities needed for these computations are the estimate of 



personal development participation rate from the NHES:95 (Po p dr = 20 percent), the percentage of 

nonparticipants reporting personal development courses in the intensive reinterview (20 of 115), and 
the percentage of participants who reported taking personal development courses for the first time in 
the intensive reinterview (14 of 91). The estimated bias for the personal development course 
participation rate is 14 percent with an estimated standard error of 3 percent. The bias-corrected point 
estimate for the personal development participation rate is 34 percent and the corresponding 95 percent 
confidence interval for this estimate is from 30 percent to 40 percent. The estimated biases are 
summarized in Table 6. 



Table 6. —Estimates of bias in overall, work-related, and personal development participation rates 



Type of participation 


NHES:95 

estimate 


Estimated 

bias 


Sampling 

error 


Bias-corrected 

estimate 


Overall 


40% 


24% 


2.7% 


64% 


Work-related 


21 


16 


3.0 


37 


Personal development 


20 


14 


3.0 


34 



Source: U.S. Department of Education, National Center for Education Statistics, Bias Study of the National Household Education Survey, 
1995. 



These estimates show that the underreporting bias is approximately the same for both 
work-related and personal development courses. While these estimates are subject to the same caveats 
as the overall estimates of participation rates, they also have an interesting implication because of the 
difference in the wording of the questions about the two types of participation. The introduction to the 
question about work-related courses does not include specific examples, but does mention courses 
taken at work, taken somewhere else but related to work or career, and courses taken to obtain a 
license or certificate related to work or career. On the other hand, the introduction to the question 
about personal development specifically mentions courses including arts and crafts, sports or 
recreation, first aid or childbirth, Bible study, or any other course. 



One way of interpreting the approximately equal biases for the two types of participation 
is that adding examples does not improve the quality of reporting in this situation. This interpretation 
is consistent with the hypothesis that respondents develop a response paradigm in the original interview 
that includes only more formal courses. If this is true, then the addition of the specific examples may 
not cause respondents to change their paradigm. Thus, a different approach might be needed to 
address the exclusion of less formal courses in reporting. The evidence on the biases by type of 
participation suggests methods other than simple examples or probes may be necessary if more 
complete reporting of work-related and personal development courses is to be obtained in the same 
interview that contains questions about formal courses. 



4.3 Response Errors in Questions on Barriers to Participation 

The NHES:95 AE interview was generally structured so that adults who reported not 
participating in a particular type of activity were asked if they were interested in taking any courses in 
this area and, if they were interested, they were asked if they knew of any courses they could have 
taken in the last 12 months. Those adults who were interested and knew of courses were then asked a 
series of questions about obstacles or barriers that might have prevented them from participating in 
these activities. 

These questions were developed for the NHES.95 and included in an attempt to better 
understand why adults did not participate in AE activities. This understanding could be particularly 
important for developing and marketing courses for adults with low educational attainment who are 
generally those most in need of adult education services. Although the questions were field-tested 
prior to the NHES:95, the reliability of the items was uncertain. These items were also included in the 
regular Reinterview Study to estimate the consistency of the adults reporting. Despite the small sample 
size for that study, the reinterview estimates showed there was the potential for large response 
consistency problems for the barriers questions. The Reinterview Report (Brick, Wernimont, and 
Montes 1996) suggested research consider different approaches for asking these questions in future 
interviews. 



The barriers questions were also included in the Bias Study to examine the validity of the 
responses to complement the information on the consistency of responses obtained from the 
Reinterview Study. The Bias Study asked respondents to describe their responses in more detail to 
determine if there was a common understanding of the questions and response categories that was 
reliable from an analytic perspective. The results of the Bias Study for the barriers items are presented 
below, beginning with the interest and knowledge questions asked of nonparticipants. 

Interest and knowledge questions. In the intensive reinterview, 48 respondents who 
reported they were not interested in taking any courses during the original interview were asked to 
more fully explain their lack of interest. About 60 percent of the respondents (29 of 48) reported that 
they were not interested in taking courses for various reasons such as having no desire to take courses, 
having no need for it in their current jobs, or being retired from work. On the other hand, about 40 
percent of the respondents (19 of 48) explained that they were too old, had problems with health, or 
had family responsibilities that made it difficult to take courses. The latter responses could be viewed 
as barriers to participating rather than reasons why the adults were not interested in participating. 



These results show that many respondents to the original interview may have said they 
were not interested in education activities because problems or commitments prevented them from 
participating in educational activities. The objective of the interest question was to determine if the 
adult had any interest and then followup a positive response to determine what, if any, barriers 
prevented the adult from participating. The response to this question also affects which subsequent 
question are asked. Since respondents answered the interest question by considering barriers rather 
than interest alone, this response can be considered an error. There are many possible explanations of 
this type of response error. For example, adults with circumstances that persistently prevent them 
from undertaking other activities may never consider the possibility of taking AE courses. These 
adults might respond that they are not interested, but if their circumstances changed, they might be 
interested. It is probably not clear to respondents that they should respond to the interest question 
irrespective of current circumstances that might be obstacles to participating. 



Because of the importance of this series of questions for adults with low educational 
attainment levels, the responses of the eight Bias Study respondents who had not graduated from high 
school are examined in more detail. These are respondents who reported in the original interview that 



they were interested in taking courses but were not aware of any courses that they could take. 
Although the sample size is very small, the responses can provide some understanding of their 
perceptions of the knowledge question. Of these eight respondents, seven offered reasons for not 
taking courses instead of referring to knowledge of courses, and this is inconsistent with the intent of 
the questions. Other questions in the interview address interest and barriers to participation. The 
verbatim responses show that the respondents could not take courses because they felt that they had no 
need to pursue a GED, family problems or responsibilities interfered, old age, ill health, or disabilities 
prevented them, they did not have the time, or they lacked transportation. Some also reported that 
they were not interested in taking courses. 

These results are especially important because the interest and knowledge questions act as 
a filter for asking adults about barriers to participating. If a substantial percentage of the respondents 
are excluded from the barriers items because they interpret the interest and knowledge questions 
differently, then the set of adults who do respond to the barriers items may not be representative of all 
adults with an interest in participating and knowledge of courses they could have taken. Since the 
intent of the interview was to estimate the types of barriers faced by adults with an interest and 
knowledge of courses, the findings show that the denominator of the percentage is probably subject to 
significant response problems. 

The findings from the Bias Study regarding questions about interest in and knowledge of 
education activities are generally consistent with those from the Reinterview Study. Both studies 
indicate potential response problems. In fact, there is a concern that respondents may not be able to 
respond accurately to questions of this nature in an interview where they do not have the opportunity 
to think carefully about the questions. Issues such as interest or knowledge may be explored more 
effectively using qualitative methodologies, such as in-depth interviews rather than the standard 
interview methods used in the NHES:95. 

Barriers questions. The barriers questions were administered to 26 respondents who 
reported in the NHES:95 interviews that they were interested in educational activities and knew about 
activities in which they could participate, but something prevented them from participating. In the 
original interview, the respondents were asked to choose one of four main barriers: time, money or 
cost, child care, or transportation. They could also specify another main barrier to participating. 



After identifying the barrier that was the greatest obstacle to their participation, respondents were 
asked to rate more specific aspects of the main barrier. The response categories for the specific aspects 
were: major obstacle, a minor obstacle, or not an obstacle. For example, specific aspects of the cost 
barrier included "amount of tuition and fees for classes," "cost of books and supplies for classes," 
"cost of child care," and "cost of transportation." 



Table 7 presents the responses of the 26 adults who answered the main barrier question 
(i.e., what was the main thing that prevented you from taking career or job related courses?) in the 
original interview and the intensive reinterview. The barriers questions for other sections of the 
interview such as ESL and ABE/GED were included in the Bias Study but too few respondents were 
sampled to report in the tables below. Overall, 18 of the 26 respondents (69 percent) reported the 
same main barrier in both interviews. The adults reported time as the main barrier more often and cost 
less often in the intensive reinterview than in the original interview. Because of the small sample size 
and the multiple response categories for the question, the consistency of the reporting may be a useful 
indicator of the quality of the responses. If the responses to the original and intensive reinterview are 
not consistent, then there is a high potential for bias but the direction of the bias may be hard to 
establish because of the small sample size. 



Table 7. --Number of adults responding to main barriers item in the NHES:95 original interview and 
intensive reinterview 



Intensive 

reinterview 


NHES:95 interview 


Total 


Main barrier 


Time 


Cost 


Child care 


Transportation 


Other 


Total 


26 


12 


9 


1 


2 


2 


Main barrier 














Time 


18 


12 


3 


0 


2 


1 


Cost 


5 


0 


5 


0 


0 


0 


Child care 


3 


0 


1 


1 


0 


1 


Transportation 


0 


0 


0 


0 


0 


0 


Other 


0 


0 


0 


0 


0 


0 



Source: U.S. Department of Education, National Center for Education Statistics, Bias Study of the National Household Education Survey, 



In the Bias Study, respondents were asked to explain more about their choice of main 
barriers and then rate the intensity of the main barriers they identified (major, minor or no obstacle). 
Their responses revealed that they attached different meanings to words like time and cost that they 
chose as their main barriers. For example, of the 12 respondents reporting time as the main barrier in 
the intensive reinterview, some explained their choice by referring to the hours they spent at work, 
some referred to the time involved in child care, and some to the time spent with other family 
responsibilities. Similarly, when money or cost was cited as the main barrier, some respondents were 
referring to lack of income due to a disability and others were referring to lack of money for tuition. 
This ambiguity with respect to the meaning of the terms could lead to inconsistent reporting, even by 
the same individuals when asked the questions more than once. A related observation is that a large 
percentage of adults gave an ‘other’ response when asked what may have prevented them from 
participating in the original interview. In response to this question (F22) in the work-related section, 
19 percent of the respondents gave an ‘other’ response (about the same percent replied ‘other’ when 
asked in the sections of the interview dealing with ESL and ABE/GED). All of this evidence implies 
that the responses are quite likely to be subject to large response variability and may be biased. 

When the respondents were asked to rate the intensity of the main barriers (i.e., whether 
the main barrier was as a major obstacle, a minor obstacle, or not an obstacle) in the Bias Study 
(question numbers F22 and F24), their responses were very inconsistent with their original ratings. 
Only 2 of the 26 respondents gave the same ratings for the specific items in both the original interview 
and the intensive reinterview. An example of this lack of consistency can be seen by looking at the 
responses to the specific obstacle "need to take care of family duties or chores around the house," 
which was rated if time was reported as the main barrier. Table 8 shows the responses to this item for 
the 12 respondents who were asked the item in both the original and intensive reinterview. The results 
of the table show the responses were not consistent. Only 4 of the 12 respondents gave the same rating 
both times. The responses for the other specific aspects were very similar to those shown in this 
example. A full list of the discrepancies in ratings of the specific aspects of the work-related barrier 
questions is given in Appendix D. 



Table 8. -Number of adults responding to the specific time barrier on need to take care of family duties 
or chores in the NHES:95 and intensive reinterview, by response category 



Intensive reinterview 


NHES:95 interview 


Total 


Type of obstacle 


Major 


Minor 


None 


Total 


12 


3 


5 


4 


Type of obstacle 










Major 


3 


1 


1 


1 


Minor 


5 


1 


2 


2 


None 


4 


1 


2 


1 



Note: Six respondents who reported time as a main barrier in the NHES:95 but not in the intensive reinterview are excluded from this 

table. 



Source: U.S. Department of Education, National Center for Education Statistics, Bias Study of the National Household Education Survey 
1995. J 



The results from the Bias Study for these items are consistent with those from the 
Reinterview Study. In that analysis, a more lenient approach was taken by collapsing response 
categories (minor obstacle and no obstacle were collapsed into one response for one analysis and major 
and minor obstacle were collapsed for another analysis). Even with the collapsing, the questions had 
low reliability in the Reinterview Study, but the sampling errors of the estimates were large. 
Generally, adults were somewhat more consistent in reporting an obstacle (collapsing major and minor) 
rather than the alternative (collapsing minor and no obstacle). 

Both the Bias Study and the Reinterview Study show that respondents’ ratings of the 
intensity of the main barriers appear to be inconsistent. Though both studies are based on small 
samples, the degree of agreement between them strengthens the common findings. Both studies 
indicate that the approach for asking about barriers to participation needs to be reconsidered. The 
analytic objectives may be better served by other methods of collecting data on obstacles because 
simple revisions in the questionnaire wordings are not likely to improve the quality of the data. 
Alternatively, the intensity questions could be eliminated or replaced by questions that ask about the 
behaviors of adults, as suggested in the Reinterview Study. 



4.4 



Switching Participation Status 



The final objective of the Bias Study was to explore reasons that responses to the 
participation item in the screening interview were not consistent with the final participation status 
reported in the NHES:95 interview. In the NHES:95, the household respondents’ answers were used 
to assign sampling rates to the adults. Some adults who were sampled and completed the extended 
interview reported their participation status differently than was reported in the screening interview. 
Ten percent of the respondents in the NHES:95 who were sampled as participants based on responses 
to the screening item became nonparticipants based on responses to questions in the extended 
interview; 9 percent switched from nonparticipant to participant status. If it is possible to reduce this 
phenomenon of switching, then the standard errors of the estimates, especially for estimates of 
participation, can be reduced. The Bias Study goal of better understanding the reasons for switching is 
an important first step in making such improvements in future surveys. 

One reason for switching participation status could be that the household respondent 
answering the screening questions does not have the knowledge necessary to accurately report the 
participation status of the adult. This would only affect those adults sampled for the extended 
interview who did not respond to the screening interviews themselves. The adults with different 
participation status after the interview who did not answer the screening items for themselves are called 
proxy switchers. Another reason for switching was that the screening question was not as elaborate 
as the full set of participation items in the extended interview and respondents may not have fully 
understood what was meant by participation. Adults who answered both the screening and extended 
interviews and still changed participation status are called self-switchers. Both of these types of 
switching were investigated in the Bias Study. 

Proxy reporting of participation status. The Bias Study included 34 "proxy switchers," 
those whose participation status changed after another household member responded to the screening 
interview. It is important to realize that the extended interviews in the NHES:95 had to be completed 
by the sampled adult; no other adult could complete this interview. The sampled adults classified as 
proxy switchers were asked to assess the knowledge of the person who answered the screening 
interview with respect to the educational activities of the sampled adult at the time of the NHES:95 
interview. They were also asked to evaluate their ability to report on the educational activities of other 
adult members of their household. 



Table 9 shows that nearly two-thirds of the sampled adults said that the screener 
respondents should have known about their educational activities and, by implication, could have 
answered the question correctly if they understood it. The sampled adult’s opinion of the knowledge 
of the household respondent did not differ dramatically for those adults that were finally classified as 
participants and for those classified as nonparticipants. (It should be noted that because the scheme 
used in the Bias Study sampled adults separately by their final participation status, row percentages 
should not be computed for this table.) The 12 sampled adults who believed the screener respondents 
would not be able to accurately report the educational activities of the other household members 
reported that the screener respondents might not know what other members of the household do at 
work and would be forced to guess to answer the participation questions. The 12 month recall period 
may also attribute to the difficulty of responding accurately for other household members. 



Table 9. -Number of "proxy switchers," by their final participation status and their opinion of the 
proxy's knowledge of their educational activities 



Adult’s opinion of 
proxy’s knowledge 


Total 


Final NHES:95 participation status 


Participant 


Nonparticipant 


Number 


Percent 


Number 


Percent 


Number 


Percent 


Total 


34 


100.0% 


11 


100.0% 


23 


100.0% 


Knowledgeable 


22 


64.7 


6 


54.5 


16 


69.6 


Not knowledgeable 


12 


35.3 


5 


45.5 


7 


30.4 



Source: U.S. Department of Education, National Center for Education Statistics, Bias Study of the National Household Education Survey 
1995. 



When the "proxy switchers" were asked whether they would know about the educational 
activities of the other members of their households, nearly three-quarters said that they would know 
about the educational activities of other household members. This question was followed by additional 
queries to evaluate whether they would need examples, would guess, or would say that they did not 
know when asked this question. Most respondents said that they would say that they did not know, 
and only a few said they would guess. 



An intriguing finding was the fact that half the respondents said they thought additional 
examples of educational activities would help them to respond accurately, despite admitting that they 
would still not know whether other household members participated in AE activities or not. This may 
be a case in which the respondents were trying to suggest ways to improve the survey in general, 
without reference to their specific circumstances. It is unclear from their responses how additional 
examples would improve the accuracy of their reports. 

On the other hand, adding examples does have some negative effects. Some respondents 
might assume that the examples are the entire list of eligible activities and ignore other types of 
activities they would otherwise report. Another disadvantage of adding examples is that it lengthens 
the time of the screening interview and this is costly because it must be done for every member of the 
household, not just those who are sampled. In the NHES:91 screener, a longer and more detailed list 
of educational activities was used, and considerable respondent annoyance was reported by 
interviewers and observed in monitoring. 



Self reporting of participation status. The Bias Study included 37 "self-switchers" (20 
self-switchers who became participants and 17 self-switchers who became nonparticipants). These are 
respondents who answered one way in the screening interview and the other way in the extended 
interview. The self-switchers were asked to explain what had happened to change their minds. 

Some of the Bias Study self-switchers who became participants in the course of the 
extended interview reported that they did not realize that courses at work or personal development 
courses were to be included as education activities. They said they were thinking about more 
traditional, structured types of education. Some of the Bias Study self-switchers who became 
nonparticipants in the NHES:95 extended interview gave rather ambiguous reasons for switching. For 
example, some said they were distracted during the screening interview, while others could not 
remember why they had changed their minds. 



The Bias Study responses on switching participation status do indicate that some proxy 
respondents may not have the knowledge needed to report accurately the educational activities of other 
household members, particularly for work-related and personal development courses. Perhaps even 
more important than the knowledge issue is the perception of the types of activities to be included 
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when respondents are asked about education activities. This appears to be a problem that affects the 
responses in the screening item. While it is possible that reporting some types of activities would be 
improved by adding examples or having a more detailed series of questions, this approach has other 
disadvantages that may outweigh any gains in variance reduction of the estimates associated with better 
reporting for the sampling of adults. Attempts to modify the screening item in this way are not 
recommended unless they are supported by research showing the decreases in variance of the estimates 
are sufficiently large to offset the additional cost in screening and the potential bias due to under- 
reporting of other types of activities. 



5. Summary and Recommendations 

The Bias Study addressed four specific objectives and uncovered some very important 
results that have implications for the analysis of the NHES:95 and the development of future surveys 
of Adult Education. The most important findings are summarized below for each of the Bias Study 
objectives along with recommendations for future AE surveys. Methodological recommendations for 
undertaking other intensive reinterviews are also presented. 

Overall participation rate. The estimated bias in the overall participation rate of adults 
was 24 percent and the bias-corrected estimate is that 64 percent of adults participated in AE in 1995. 
This is substantially larger than the 40 percent reported in the NHES:95. This estimate was based on 
the assumption that the nonparticipant sample for the Bias Study has the same distribution as that for 
all nonparticipants in the NHES:95. Even though the assumption does not hold exactly, the findings 
show that participation was underreported in the NHES:95, and the underreporting could be very 
substantial. 



It was hypothesized that a reason for the underreporting may have been due to how 
respondents reacted to the context of the NHES:95 interview. Since the interview began by asking 
about formal types of participation, some respondents may have created a response paradigm that 
restricted their answers to more formal courses when the work-related and personal development 
courses were asked. 



The evidence from the Bias Study indicates the estimates of participation are less than 
they would be if adults completely understood the intent of the survey. However, it is recommended 
that the NHES:95 participation rates be published without adjustments. The main reason for this 
recommendation is that the bias-corrected estimates are not very reliable themselves because they are 
based on small samples and assumptions about their sampling distribution. Any estimates or 
adjustments based on the Bias Study estimates would be very unstable. 

For future studies, a new approach to asking about less formal courses is suggested. One 
idea for an approach that should be considered is proposed after reviewing the findings for work- 
related and personal development course participation rates. 

Participation in work-related and personal development courses. The underreporting 
of participation for work-related and personal development courses was substantial and about the same 
size for each of these types of participation. The bias-corrected estimate of the percentage of adults 
participating in work-related courses was 37 percent rather than the 21 percent estimated from the 
NHES:95. The bias-corrected estimate for the personal development participation rate was 34 percent 
rather than the 20 percent estimated from the NHES:95. 

Despite the fact that more examples were used for the personal development courses than 
for the work-related course question, the estimated biases were approximately the same for the two 
types of participation. This suggests that simply adding examples to the wordings of the questions may 
not improve the quality of reporting and that other approaches to the underreporting problem may be 
needed. 



A new interviewing approach is proposed that makes the respondent aware of a change in 
the nature of the interview prior to asking about work-related participation. If the adults have 
developed a response paradigm that focuses on formal types of participation (i.e., traditional schooling 
or formal programs), then a relatively drastic intervention may be needed to modify this behavior. 
More traditional methods of providing examples and probes might have little impact. Different types 
of intervention should be considered. For example, a simple statement that the interview will now 
focus on less formal types of activities may not be sufficient. However, a modification in which the 
respondents are asked to actively cooperate in changing the focus, for example, by asking them to give 



examples of less formal courses, might be more effective. Clearly, cognitive research on this topic is 
needed before any interviewing method is implemented. 

Barriers to participation. The adults were not very consistent in responding to the items 
about interest in and knowledge of specific AE activities. When asked to explain their reasons for lack 
of interest, about 40 percent of the respondents mentioned obstacles rather than reasons they were not 
interested in participating. When adults who were interested but not aware of courses that they could 
take were asked for explanations, they too offered reasons for not taking courses instead of referring to 
their knowledge of courses. Since these questions act as a filter for asking adults about barriers to 
participating, the inconsistency and lack of validity of responses to these questions results in excluding 
some adults from being asked subsequent questions reducing the value of these questions for analysis. 

In addition, the Bias Study confirmed the Reinterview Study findings with respect to the 
consistency of the responses for the barriers questions. The responses to the main barrier items were 
reported the same in the original interview and the intensive reinterview less than 70 percent of the 
time. Adults also ascribed different meanings to the words used to describe the main barriers. The 
ratings of the intensity of the main barriers appeared to be inconsistent. 

When using the NHES:95, we recommend that analysts be urged to inform readers of the 
consistency problems associated with the barriers questions. While it is reasonable to report on the 
main barriers items with this caveat, we recommend that analysts avoid the ratings of the intensity of 
the main barriers. This is also a recommendation in the Reinterview Study. 

For future studies, the approach to asking about barriers needs to be reconsidered, 
especially for the intensity measures. In-depth interviews should be considered instead of including 
these items in future general telephone surveys. Possible approaches include dropping the intensity 
questions entirely or replacing them with questions that are more related to the behaviors of adults, as 
suggested in the Reinterview Study. Clearly, a research agenda needs to be established for these 
questions if a more effective measure of barriers is to be collected from large-scale telephone surveys. 

Switching rates. The Bias Study explored the reasons for inconsistent reporting of 
participation status in the NHES:95, the switching problem, but did not uncover many features that 



were not already known. Some proxy respondents may not have the knowledge needed to report the 
educational activities of other household members accurately. Furthermore, it appears that many 
adults do not understand the full range of activities that should be included when the screening 
questions are asked. However, the Bias Study did not lead to ideas for new and efficient ways to 
improve this situation. 

Methodological recommendations. The intensive reinterview methodology applied in 
the Bias Study appears to have good potential as a method for detecting biases, especially if more 
traditional methods like record check studies are not feasible. Despite the fact that this was the first 
effort at doing this type of survey in the NHES, the results were promising. Underreporting was 
discovered and it appears that response biases can be estimated using this methodology. The 
alternative approach of using reconciled reinterviews has not proven to be successful for estimating 
bias. 



There are a number of potential improvements that were recognized based on the 
experiences of this study. One of the most important recommendations is that the sample design 
should be a valid probability sample with a large enough sample size so that reliable estimates of biases 
and their standard errors can be made. In this study, the sample size was small, the sample was 
targeted to meet specific goals, and the results cannot be generalized to the entire population without 
making assumptions about the distribution of the sample. This approach was deemed appropriate for 
the first attempt at this type of study, but it has obvious limitations for analysis. 

Another area that can be improved is the selection of issues that should be addressed 
using this method. For estimating biases in the AE participation rates, the intensive reinterview 
approach was very effective. However, for estimating the consistency of reporting, such as in the 
switching participation questions, the standard reinterview is a better mechanism. It is specifically 
designed to do this and the Bias Study often simply confirmed the findings of the Reinterview Study 
without adding much new information. 

This Bias Study also attempted to explore why the respondents replied in certain ways for 
the questions about interest in and knowledge of AE courses. This was not very successful. In 
general, respondents do not articulate reasons very well in interviews, even if the interviews are more 



conversational and unstructured like the intensive reinterview. Other approaches, such as focus group 
methods, are better suited for this purpose. 

Finally, from an operational perspective, it is important to understand that this method is 
more costly than a regular reinterview because it requires more effort to organize than the standard 
interview or reinterview. To be effective, interviewers must be trained extensively because the method 
requires nonstandard approaches to interviewing. If interviewers are not used, staff who are already 
trained in cognitive methods should be used, and this may be expensive if the sample size is large. As 
a result, we recommend that this method be used primarily when there is an indication of reporting 
errors that might result in biases and the estimates subject to the biases are important to the survey 
objectives. 
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Appendix A : 



Adult Education Bias Study Protocol 



» 



BIAS STUDY 



Date of original interview 
Time of original interview 



CASE NUMBER 
TEL. NUMBER 



STATE 



TIME ZONE: E C M P 



NAME OF RESPONDENT 

STRATUM MAINRSLT _ 

□ Participant [Number of courses 



IBGRADE 



□ 

□ 



□ 


Work-related 


□ 


Apprenticeship 


□ 


Other formal 


□ 


College or graduate degree 


□ 


ESL 


□ 


Vocational technical program 


□ 


ABE/GED 







Does not know 



Nonparticipant 
Not interested □ 

□ ESL 

□ ABE/GED 

□ Work-related 
Barriers Questions 

□ ESL 

□ ABE/GED 

□ Work-Related 

Switcher [to participant/nonparticipant status] 

□ Proxy [Screener respondent 

□ Self 



May I speak with (respondent name)? Hello, my name is (interviewer name) and I'm calling for 
) the U.S. Department of Education. Recently, you and many other people across the country completed 

an interview about education activities, and we are calling back some people to see if we can improve 
our survey. Your help will be greatly appreciated, and it will take only a few minutes. 







c: f 

OJL 



A-l 



Date of original interview 



RECALL 

FOR PARTICIPANTS: 

When we spoke with you, we asked you whether or not you had participated in a variety of courses, 
workshops, or seminars in the 12 months before we called, that is, from (month) 94 to (month) 95. 



You reported that you had participated in (an apprenticeship program/college or graduate courses/a 
vocational program/basic skills or high school completion classes/ESL classes), [go to next page.] 



You (also) reported that you had participated in (number) courses. (They/Some of them) were: 
COURSE NAME WHEN TAKEN (MONTH/YEAR) 



1. 

2 . 

3. 

4. 

5. 

6. 

7. 

8 . 
9. 



Do you remember when you took (course 1)? [if the exact month and year cannot be 
recalled] Was it in the spring or summer, perhaps? And that was 1994? [ask for course 2, 
ETC.] 



Date of original interview 



We know that many people forget about some educational activities when asked in an interview or 
can't easily recall exactly when they did these things. 

We’d like to know if you now remember any other classes or courses with an instructor that you took 
during the 12 months before we called you. We want to make sure that we made it clear that we were 
interested in all kinds of educational activities for work or personal reasons, so let me give you a few 
examples: 

[USE EXAMPLES THAT ARE DIFFERENT FROM THE TYPE OF COURSES LISTED ABOVE.] 

Crafts or cooking CPR word processing 

physical fitness or aerobics stress management diversity training 

sports lessons like tennis or golf time management communication skills 

Bible study childbirth classes 

[IF APPROPRIATE] Could you think back for a moment to [use the respondent's time reference 
(E.G., MONTHS, SEASONS) TO PROBE THE TIME PERIODS NOT MENTIONED PREVIOUSLY], Were you 
doing other educational activities then? 



NEW COURSE NAME 



WHEN TAKEN (MONTH/YEAR) 



Date of original interview 



TYPES OF EDUCATIONAL ACTIVITIES 



NONPARTICIPANTS: 

When we spoke with you, we asked you whether or not you had participated in a variety of courses, 
workshops, or seminars in the 12 months before we called, that is, from (month) 94 to (month) 95, 
and you said that you had not. 

We know that many people are not involved in educational activities, but sometimes they did take a 
class and just didn’t recall it when asked in an interview. 

We’d like to know if you now remember any classes or courses with an instructor that you took during 
the 12 months before we called you. We want to make sure that we made it clear that we were 
interested in all kinds of educational activities for work or personal reasons, so let me give you a few 
examples: 

[USE a few examples from each column or others that might be meaningful to the 

RESPONDENT.] 

Crafts or cooking CPR word processing 

physical fitness or aerobics stress management diversity training 

sports lessons like tennis or golf time management communication skills 

Bible study childbirth classes 



[IF appropriate] Could you think back for a moment to [refer to a range of months, seasons, 
or intervals between holidays to probe 1 or 2 time periods.] Were you doing educational 
activities then? 



NEW COURSE NAME 



WHEN TAKEN (MONTH/YEAR) 



Date of original interview 



NOT INTERESTED 



When we first talked to you, you said you weren't really interested in taking (ESL classes/basic skills 
or high school completion classes/work-related courses). I'm wondering if you could explain a bit 
more about that. 



[EXAMPLES:] What came to mind when we asked if you were interested? 



Were you ever interested in taking some courses? 



NOT AWARE 



When we first talked to you, you said you didn't know about any (ESL classes/basic skills or high 
school completion classes/work-related). I'm wondering if you could explain a bit more about that. 



[EXAMPLE:] Have you ever tried to find out about courses you could take? 



Did you ever think about doing that? 



Date of original interview 



BARRIERS 

NONPARTICIPANTS: 



When we first talked to you, you said you were interested in taking (basic skills or high school 
completion classes/ESL classes/work-related courses), but some things prevented you from doing this. 



What was the main thing that prevented you from participating? 



Was there anything else that made it hard for you to take courses? 



[IF ONE OBSTACLE WAS CLEARLY THE MOST IMPORTANT, GO TO NEXT PAGE. IF TWO OR MORE THINGS 

seemed to be equally important IN preventing participation:] What was the main thing that 
prevented you? 



Now, I’d like to focus a little more on some (time/cost/child care/transportation) related problems that 
may have prevented you from taking courses. 



If time was the major obstacle, go to A. 

If money or cost was the major obstacle, go to B. 

If child care was the major obstacle, go to C. 

If transportation was the major obstacle, go to D. 



Date of original interview 



A. Could you explain a little more about why time was a problem? 



[INDICATE WHICH OF THE FOLLOWING WAS MENTIONED BY THE RESPONDENT AND TO WHAT EXTENT 
IT SEEMED TO BE AN OBSTACLE. PROBE IF APPROPRIATE.] 

Major Minor Not N/A 

Desire to spend time with family 1 2 3 4 

Need to take care of family duties or chores 

around the house 1 2 3 4 



Classes only offered during work hours 



1 



2 3 4 



Work responsibilities prevent taking courses 

either during work hours or after work hours 1 2 3 4 



Activities outside of work conflict with the time 

a class is offered 1 2 3 4 



Time it takes to travel to and from class 



1 



2 3 4 



B. Could you explain a little more about why cost was a problem? 



[INDICATE WHICH OF THE FOLLOWING WAS MENTIONED BY THE RESPONDENT AND TO WHAT EXTENT 
IT SEEMED TO BE AN OBSTACLE. PROBE IF APPROPRIATE.] 

Major Minor Not N/A 

Amount of tuition and fees for classes 1 2 3 4 

Cost of books and supplies for classes 1 2 3 4 

The cost of child care 1 2 3 4 

The cost of transportation 1 2 3 4 



Date of original interview 

C. Could you explain a little more about why child care was a problem? 



« 



« 



[INDICATE WHICH OF THE FOLLOWING WAS MENTIONED BY THE RESPONDENT AND TO WHAT EXTENT 



IT SEEMED TO BE AN OBSTACLE. PROBE IF APPROPRIATE.] 


Major 


Minor 


Not 


N/A 


The cost of child care 


1 


2 


3 


4 


The availability of child care 




2 


3 


4 



D. Could you explain a little more about why transportation was a problem? 



[INDICATE WHICH OF THE FOLLOWING WAS MENTIONED BY THE RESPONDENT AND TO WHAT EXTENT 
IT SEEMED TO BE AN OBSTACLE. PROBE IF APPROPRIATE.] 

Major Minor Not N/A 



The cost of transportation 



1 



2 3 4 



The availability of transportation 



1 



2 3 4 



The time it would take to travel to and 

from courses 1 2 3 



% 




DO 



A-8 



Date of original interview 



PROXY REPORTING OF PARTICIPATION STATUS 
PROXY SWITCHERS: 

When we called last time, (screener name) answered a few questions about each member of your 
household and reported that you (did/did not) participate in educational activities in the past 12 
months. However, when we talked to you, you said that you (did/did not) participate. 

We want to know if we can improve our question to collect more accurate information about other 
members of the household. 

The question we asked (SCREENER NAME) was, "During the past 12 months, did (RESPONDENT NAME) 
take classes, programs, courses, workshops, or seminars of any kind for any reason?" 

1. Do you think (SCREENER NAME) knew about your educational activities when we asked the 
question? 

YES 1 

[if yes, ask:] Do you have any idea why (SCREENER NAME) said you (had/had 
not) participated? 

RESPONSE: 

NO 2 



2. If I asked you whether other adults in your household participated or did not participate in 
classes or other education activities in the past 12 months, do you think you would know the 
right answer without checking with the person? [code both yes and no, if applicable.] 

YES 1 



[FOR A SPECIFIC HOUSEHOLD MEMBER, SPECIFY WHO] 

NO 2 

[IF NO FOR ANY HOUSEHOLD MEMBER, ASK AND CODE ALL THAT APPLY:] 
Do you think you... 

Would need some examples of educational 



activities to answer, 1 

You would probably guess, or 2 

You would say you did not know? 3 



Can you think of any ways we could improve our question? 



RESPONSE: 



Date of original interview 



CHANGES IN SELF-REPORT OF PARTICIPATION STATUS 

SELF-SWITCHERS: 



When we called you before, first we asked a question about educational activities. It was: "During the 
past 12 months, did you take classes, programs, courses, workshops, or seminars of any kind for any 
reason?" You told us at first that you (did/did not) participate in educational activities in the past 12 
months. 

Then we asked about different types of educational activities that you might have participated in, and 
you said that you (did/did not) participate in (one or more/any) of them. 

Could you tell us what happened to change your mind? 



[FOR THOSE WHO switched to participation status] Was there something in particular that we said 
that helped you remember? 



Can you think of any ways we could improve our question? 



Appendix B: 



Training Guide 



BIAS STUDY TRAINING AGENDA 



Overview 

a. Brief explanation of the purpose of the study and the interviewing approach 
— Improve questionnaire to collect better data ; 

— Use conversational format to capture the perspective of the respondent ; 

— Employ some specific techniques to see of there are better ways to get the information 

we are interested in 

b. Format of the training session 

— Conversational; all should participate . 



Background 

a. The NHES :95 Adult Education component 

— They are familiar; have instrument and manual; have heard interviews; 

— Recap major research purpose: How many adults participate in educational 

activities , what types of activities do they participate in (especially work-related ) f 
what keeps people who want to participate in educational activities from taking part? 

— Questions about education/work experience/language background; participation in or 
barriers to participation in ESL t ABE/GED, credential apprenticeship , work-related \ 
other formal computer-only/interactive; background information 

Purpose of the bias study 

Goal 1: To obtain a more accurate count of the type of courses taken during the 12 months 

preceding the original interview 

a. The issues of telescoping and difficulty of recalling participation in educational activities 

— Spur of the moment request for information; better info by asking about specific types; 

— Recall over 12 months— telescoping 

— Want to know if some types of courses are over-reported 

b. Population: participants in any type of educational activity-ESL, ABE/GED, credential 
(college, voc.ed), apprenticeship, work-related, other formal. 



Goal 2: To see if respondents recall courses now that they did not recall when we first 

spoke with them. 

a. Underreporting participation in educational activities during an interview because respondents 
were uncertain what we meant by educational activities 

— Giving them examples of types of courses may help them recall 

— Also will use a technique to help them think back over the past 12 months 

b. Population: nonparticipants and participants 

- Participants in some types may have forgotten other types . 

Goal 3: To understand better why low-education adults have no interest in taking courses, 

do not know about courses they could take, or have other obstacles that prevent 
them from taking courses 

a. The interest question 

— Can they articulate why? 

b. The knowledge question 

— Have they ever tried to find out? 

c. The accuracy of the barriers to participation that were asked in the interview 
— What prevents people who want to participate from participating? 

— We want the respondent's own point of view , not influenced by ours . 

d. Population: participants and nonparticipants, but only low-education respondents 

Goal 4: To examine reasons for switching participation status 

a. "Proxy switchers" and the accuracy of reporting for other household members 

— Why might people know or not know about the educational participation of other 
household members? 

b. "Self-switchers" and aids to recall 

— What cues were helpful to their recall? 

c. Population: participants and nonparticipants who switched status 



Methodology: the semi-structured interview 



- Balance between strict adherence to questions (same context) and a more conversational 
mode; 

- Quantitative vs. qualitative; 

- Using the more qualitative approach to assess the quality of the data gathered in the more 
quantitative approach; 

- Our data quality will depend directly on our understanding of the research goals and our skill 
in eliciting the information required; 

- Each topic in the protocol must be addressed [ but followup probes will vary with the 
situation; 

- Use of open ended probes: 

"Could you tell me a little more about that?” 

”Td be interested in your own words to describe that . " 

"Is there anything else you'd like to add?” 

- Listening actively to the respondent; giving feedback when appropriate; 

- Creative use of silences. 

- Recording information : key words; maintaining interviewer-respondent interaction . 

Overview of the protocol 

- Constructed to achieve goals; customized according to participant. 

a. The timeline 

— A tool with multiple uses. 

b. Section-by-section through the protocol 

— Generating probes 

— Recording responses 

Role plays 

a. Participant (high ed) in work related and other formal 

b. Nonparticipant (low ed), barriers to work-related 

c. Participant in ABE/GED, no interest in work related, proxy-switcher 

d. Nonparticipant (low ed), barriers to ESL, self-switcher 

e. Participant (low ed) in apprenticeship, no interest in ABE/GED 

f. Nonparticipant (low ed), no knowledge of ESL, proxy switcher 



Appendix C: 



Interviewer Handout 



BIAS STUDY 



Goal 1: To obtain a more accurate count of the type of courses taken during the 12 months 
preceding the original interview 



Goal 2: To see if respondents recall courses now that they did not recall when we first spoke with 

them. 



Goal 3: To understand better why low-education adults have no interest in taking courses, do not 
know about courses they could take, or have other obstacles that prevent them from 
taking courses 



Goal 4: To examine reasons for switching participation status 



Methodology: the semi-structured interview 

Balance between strict adherence to questions (same context) and a more 
conversational mode 

Quantitative vs. qualitative 

Using the more qualitative approach to assess the quality of the data gathered in the 
more quantitative approach 

Our data quality will depend directly on our understanding of the research goals and 
our skill in eliciting the information required 

Each topic in the protocol must be addressed, but followup probes will vary with the 

situation 

Use of open ended probes: 

"Could you tell me a little more about that?" 

"I'd be interested in your own words to describe that. " 

"Is there anything else you'd like to add?" 

Listening actively to the respondent; giving feedback when appropriate 

Creative use of silences 

Recording information: key words; maintaining interviewer-respondent interaction 



Appendix D: 



Responses to Barrier Questions 



CASE 1 



Main barrier 



Specific aspects of the main barrier 

Desire to spend time with family 
Take care of family duties or chores 
Classes only offered during work hours 
Work responsibilities prevent taking 
courses either during work hours or 
after work hours 

Activities outside of work conflict with 
the time a class is offered 
Time it takes to travel to and from class 



NHES:95 


Bias Study 


Time 


Time 


Major obstacle 
Minor obstacle 
Minor obstacle 


Major obstacle 
Not an obstacle 
Not an obstacle 


Not an obstacle 


Not an obstacle 


Minor obstacle 
Not an obstacle 


Not an obstacle 
Not an obstacle 



CASE 2 



Main barrier 



NHES:95 Bias Study 

Time Time 



Specific aspects of the main barrier 

Desire to spend time with family 
Take care of family duties and chores 
Classes only offered during work hours 
Work responsibilities prevent taking 
courses either during work hours or 
after work hours 

Activities outside of work conflict with 
the time a class is offered 
Time it takes to travel to and from class 



Minor obstacle 
Not an obstacle 
Minor obstacle 



Major obstacle 

Minor obstacle 
Minor obstacle 



Not an obstacle 
Major obstacle 
Not an obstacle 



Major obstacle 

Not an obstacle 
Not an obstacle 



CASE 3 



Main barrier 



Specific aspects of the main barrier 

Desire to spend time with family 
Take care of family duties or chores 
Classes only offered during work hours 
Work responsibilities prevent taking 
courses either during work hours or 
after work hours 

Activities outside of work conflict with 
the time a class is offered 
Time it takes to travel to and from class 



NHES:95 


Bias Study 


Time 


Time 


Minor obstacle 
Not an obstacle 
Major obstacle 


Minor obstacle 
Minor obstacle 
Major obstacle 


Major obstacle 


Major obstacle 


Not an obstacle 
Minor obstacle 


Major obstacle 
Not an obstacle 



CASE 4 



Main barrier 



NHES:95 Bias Study 

Time Time 



Specific aspects of the main barrier 

Desire to spend time with family 
Take care of family duties and chores 
Classes only offered during work hours 
Work responsibilities prevent taking 
courses either during work hours or 
after work hours 

Activities outside of work conflict with 
the time a class is offered 
Time it takes to travel to and from class 



Major obstacle 
Major obstacle 
Not an obstacle 



Not an obstacle 

Major obstacle 
Not an obstacle 



Major obstacle 
Minor obstacle 
Not an obstacle 



Not an obstacle 

Major obstacle 
Not an obstacle 



CASE 5 



Main barrier 



NHES:95 Bias Study 

Time Time 



Specific aspects of the main barrier 

Desire to spend time with family 
Take care of family duties or chores 
Classes only offered during work hours 
Work responsibilities prevent taking 
courses either during work hours or 
after work hours 

Activities outside of work conflict with 
the time a class is offered 
Time it takes to travel to and from class 



Major obstacle 
Major obstacle 
Not an obstacle 



Not an obstacle 

Minor obstacle 
Not an obstacle 



Major obstacle 
Major obstacle 
Minor obstacle 



Not an obstacle 

Not an obstacle 
Not an obstacle 



CASE 6 



Main barrier 



Specific aspects of the main barrier 

Desire to spend time with family 
Take care of family duties and chores 
Classes only offered during work hours 
Work responsibilities prevent taking 
courses either during work hours or 
after work hours 

Activities outside of work conflict with 
the time a class is offered 
Time it takes to travel to and from class 



NHES:95 


Bias Study 


Cost 


Time 


Inapplicable 

Inapplicable 

Inapplicable 


Not an obstacle 
Not an obstacle 
Major obstacle 


Inapplicable 


Major obstacle 


Inapplicable 

Inapplicable 


Not an obstacle 
Not an obstacle 



CASE 7 





NHES:95 


Bias Study 


Main barrier 


Other barrier 


Time 


Specific aspects of the main barrier 






Desire to spend time with family 
Take care of family duties or chores 
Classes only offered during work hours 
Work responsibilities prevent taking 
courses either during work hours or 


Inapplicable 

Inapplicable 

Inapplicable 


Not an obstacle 
Not an obstacle 
Not an obstacle 


after work hours 

Activities outside of work conflict with 


Inapplicable 


Major obstacle 


the time a class is offered 
Time it takes to travel to and from class 


Inapplicable 

Inapplicable 


Not an obstacle 
Not an obstacle 



CASE 8 



Main barrier 



Specific aspects of the main barrier 

Desire to spend time with family 
Take care of family duties and chores 
Classes only offered during work hours 
Work responsibilities prevent taking 
courses either during work hours or 
after work hours 

Activities outside of work conflict with 
the time a class is offered 
Time it takes to travel to and from class 



NHES:95 


Bias Study 


Time 


Time 


Not an obstacle 
Not an obstacle 
Not an obstacle 


Not an obstacle 
Not an obstacle 
Not an obstacle 


Major obstacle 


Not an obstacle 


Not an obstacle 
Not an obstacle 


Not an obstacle 
Not an obstacle 



CASE 9 



Main barrier 



Specific aspects of the main barrier 

Desire to spend time with family 
Take care of family duties or chores 
Classes only offered during work hours 
Work responsibilities prevent taking 
courses either during work hours or 
after work hours 

Activities outside of work conflict with 
the time a class is offered 
Time it takes to travel to and from class 



NHES:95 


Bias Study 


Transportation 


Time 


Inapplicable 

Inapplicable 

Inapplicable 


Not an obstacle 
Not an obstacle 
Major obstacle 


Inapplicable 


Major obstacle 


Inapplicable 

Inapplicable 


Not an obstacle 
Not an obstacle 



CASE 10 



Main barrier 



NHES:95 Bias Study 

Time Time 



Specific aspects of the main barrier 

Desire to spend time with family Not an obstacle 

Take care of family duties and chores Minor obstacle 

Classes only offered during work hours Not an obstacle 

Work responsibilities prevent taking 
courses either during work hours or 
after work hours Not an obstacle 

Activities outside of work conflict with 

the time a class is offered Major obstacle 

Time it takes to travel to and from class Not an obstacle 



Minor obstacle 
Major obstacle 
Not an obstacle 



Not an obstacle 

Not an obstacle 
Not an obstacle 



CASE 11 



« 





NHES:95 


Bias Study 


Main barrier 


Transportation 


Time 


Specific aspects of the main barrier 


Desire to spend time with family 


Inapplicable 


Not an obstacle 


Take care of family duties or chores 


Inapplicable 


Not an obstacle 


Classes only offered during work hours 
Work responsibilities prevent taking 
courses either during work hours or 


Inapplicable 


Not an obstacle 


after work hours 

Activities outside of work conflict with 


Inapplicable 


Major obstacle 


the time a class is offered 


Inapplicable 


Not an obstacle 


Time it takes to travel to and from class 


Inapplicable 


Not an obstacle 






CASE 12 





NHES:95 


Bias Study 


Main barrier 


Cost 


Time 


Specific aspects of the main barrier 


Desire to spend time with family 


Inapplicable 


Not an obstacle 


Take care of family duties and chores 


Inapplicable 


Not an obstacle 


Classes only offered during work hours 
Work responsibilities prevent taking 
courses either during work hours or 


Inapplicable 


Major obstacle 


after work hours 

Activities outside of work conflict with 


Inapplicable 


Major obstacle 


the time a class is offered 


Inapplicable 


Not an obstacle 


Time it takes to travel to and from class 


Inapplicable 


Not an obstacle 
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CASE 13 



Main barrier 



Specific aspects of the main barrier 

Desire to spend time with family 
Take care of family duties or chores 
Classes only offered during work hours 
Work responsibilities prevent taking 
courses either during work hours or 
after work hours 

Activities outside of work conflict with 
the time a class is offered 
Time it takes to travel to and from class 



NHES:95 


Bias Study 


Time 


Time 


Minor obstacle 
Minor obstacle 
Major obstacle 


Major obstacle 
Minor obstacle 
Major obstacle 


Major obstacle 


Major obstacle 


Minor obstacle 
Minor obstacle 


Major obstacle 
Major obstacle 



CASE 14 



Main barrier 



NHES:95 Bias Study 

Time Time 



Specific aspects of the main barrier 

Desire to spend time with family Major obstacle 

Take care of family duties and chores Minor obstacle 

Classes only offered during work hours Major obstacle 

Work responsibilities prevent taking 
courses either during work hours or 
after work hours Major obstacle 

Activities outside of work conflict with 

the time a class is offered Minor obstacle 

Time it takes to travel to and from class Major obstacle 



Not an obstacle 
Not an obstacle 
Major obstacle 



Minor obstacle 

Not an obstacle 
Not an obstacle 



CASE 15 



Main barrier 



NHES:95 Bias Study 

Time Time 



Specific aspects of the main barrier 

Desire to spend time with family 
Take care of family duties or chores 
Classes only offered during work hours 
Work responsibilities prevent taking 
courses either during work hours or 
after work hours 

Activities outside of work conflict with 
the time a class is offered 
Time it takes to travel to and from class 



Major obstacle 
Major obstacle 
Major obstacle 



Major obstacle 

Minor obstacle 
Not an obstacle 



Not an obstacle 
Not an obstacle 
Minor obstacle 



Not an obstacle 

Major obstacle 
Not an obstacle 



CASE 16 



Main barrier 



Specific aspects of the main barrier 

Desire to spend time with family 
Take care of family duties and chores 
Classes only offered during work hours 
Work responsibilities prevent taking 
courses either during work hours or 
after work hours 

Activities outside of work conflict with 
the time a class is offered 
Time it takes to travel to and from class 



NHES:95 Bias Study 

Cost Time 



Inapplicable 


Not an obstacle 


Inapplicable 


Not an obstacle 


Inapplicable 


Major obstacle 



Inapplicable 


Major obstacle 


Inapplicable 


Not an obstacle 


Inapplicable 


Not an obstacle 



CASE 17 



Main barrier 



NHES:95 

Time 



Specific aspects of the main barrier 

Desire to spend time with family 
Take care of family duties or chores 
Classes only offered during work hours 
Work responsibilities prevent taking 
courses either during work hours or 
after work hours 

Activities outside of work conflict with 
the time a class is offered 
Time it takes to travel to and from class 



Major obstacle 
Minor obstacle 
Minor obstacle 



Not an obstacle 

Not an obstacle 
Not an obstacle 



CASE 18 



Main barrier 



NHES:95 

Time 



Specific aspects of the main barrier 

Desire to spend time with family 
Take care of family duties and chores 
Classes only offered during work hours 
Work responsibilities prevent taking 
courses either during work hours or 
after work hours Not an obstacle 

Activities outside of work conflict with 

the time a class is offered Not an obstacle 

Time it takes to travel to and from class Not an obstacle 



Not an obstacle 
Not an obstacle 
Major obstacle 



Bias Study 
Time 



Major obstacle 
Minor obstacle 
Major obstacle 



Not an obstacle 

Not an obstacle 
Minor obstacle 



Bias Study 
Time 



Not an obstacle 
Minor obstacle 
Major obstacle 



Major obstacle 

Minor obstacle 
Minor obstacle 



CASE 19 



Main barrier 



NHES:95 

Cost 



Specific aspects of the main barrier 

Amount of tuition and fees for classes 
Cost of books and supplies for classes 
The cost of child care 
The cost of transportation 



Not an obstacle 
Minor obstacle 
Inapplicable 
Major obstacle 



CASE 20 



Main barrier 



NHES:95 

Cost 



Specific aspects of the main barrier 

Amount of tuition and fees for classes 
Cost of books and supplies for classes 
The cost of child care 
The cost of transportation 



Major obstacle 
Major obstacle 
Inapplicable 
Minor obstacle 



CASE 21 



Main barrier 



NHES:95 

Cost 



Specific aspects of the main barrier 

Amount of tuition and fees for classes 
Cost of books and supplies for classes 
The cost of child care 
The cost of transportation 



Minor obstacle 
Not an obstacle 
Inapplicable 
Minor obstacle 



Bias Study 
Cost 



Major obstacle 
Minor obstacle 
Inapplicable 
Not an obstacle 



Bias Study 
Cost 



Major obstacle 
Major obstacle 
Inapplicable 
Major obstacle 



Bias Study 
Cost 



Major obstacle 
Major obstacle 
Inapplicable 
Not an obstacle 



CASE 22 



Main barrier 



NHES:95 

Cost 



Specific aspects of the main barrier 

Amount of tuition and fees for classes 
Cost of books and supplies for classes 
The cost of child care 
The cost of transportation 



Major obstacle 
Major obstacle 
Not an obstacle 
Not an obstacle 



CASE 23 



Main barrier 



NHES:95 

Cost 



Specific aspects of the main barrier 

Amount of tuition and fees for classes 
Cost of books and supplies for classes 
The cost of child care 
The cost of transportation 



Major obstacle 
Major obstacle 
Inapplicable 
Minor obstacle 



CASE 24 



NHES:95 

Main barrier Cost 

Specific aspects of the main barrier 

The cost of child care Inapplicable 

The availability of child care Inapplicable 



Bias Study 
Cost 



Major obstacle 
Major obstacle 
Not an obstacle 
Not an obstacle 



Bias Study 
Cost 



Major obstacle 
Major obstacle 
Inapplicable 
Major obstacle 



Bias Study 
Child care 



Not an obstacle 
Not an obstacle 



CASE 25 



Main barrier 



NHES:95 
Child care 



Specific aspects of the main barrier 

The cost of child care Major obstacle 

The availability of child care Not an obstacle 



CASE 26 



Main barrier 



NHES:95 
Don't know 



Specific aspects of the main barrier 

Inapplicable 
Inapplicable 



The cost of child care 
The availability of child care 





Bias Study 
Child care 

Major obstacle 
Not an obstacle 



Bias Study 
Child care 

Not an obstacle 
Not an obstacle 
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